Systems and methods for meso-dissection

ABSTRACT

The subject disclosure presents systems and methods for improved meso-dissection of biological specimens and tissue slides including importing one or more reference slides with annotations, using inter-marker registration algorithms to automatically map the annotations to an image of a milling slide, and dissecting the annotated tissue from the selected regions in the milling slide for analysis, while concurrently tracking the data and analysis using unique identifiers such as bar codes.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a continuation of International Patent Application No.PCT/EP2016/051895 filed Jan. 29, 2016, the benefit of which is claimed.Benefit is further claimed to U.S. Provisional Patent Application No.62/110,476, filed Jan. 31, 2015. The contents of these relatedapplications are incorporated by reference herein.

FIELD OF THE SUBJECT DISCLOSURE

The present subject disclosure relates to imaging for medical diagnosis.More particularly, the present subject disclosure relates to automatedmicro-dissection (meso-dissection) of tissue specimens using referenceimages.

BACKGROUND OF THE SUBJECT DISCLOSURE

In the analysis of biological specimens such as tissue sections, blood,cell cultures and the like, biological specimens are mounted on a slide,stained with one or more combinations of stain and biomarkers, and theresulting assay is viewed or imaged for further analysis. Observing theassay enables a variety of processes, including diagnosis of disease,assessment of response to treatment, and development of new drugs tofight disease. An assay includes one or more stains conjugated to anantibody that binds to protein, protein fragments, or other objects ofinterest in the specimen. Subsequent to staining, the assay may beimaged for further analysis of the contents of the tissue specimen.Further, an adjacent section of the tissue specimen from the same tissueblock may be mounted on a second glass slide, henceforth referred to asa milling slide, and specific areas of interest may be dissected forfurther analysis. For instance, dissection of slide-mounted tumorsamples is often used to enrich cancer cells in order to generate bettersignal to noise ratios in subsequent biochemical characterization. Manyclinical laboratories utilize manual dissection for practical reasonsand to avoid the expense and difficulties of laser microdissectionsystems. Unfortunately, manual methods often lack resolution and processdocumentation.

Existing slide-mounted tissue meso-dissection systems such as thosemanufactured by AvanSci Bio® (details athttp://avansci-bio.com/uploads/CDP-08_MesoDissection_System_Flyer_Rev_1.pdf)provide better precision than manual methods while also providingdigital image guidance and electronic process documentation. Ameso-dissection system may comprise a micro tissue mill that employs aspecialized disposable mill bit that simultaneously dispenses liquid,cuts tissue from the slide-mounted tissue surface, and aspirates theliquid along with the displaced tissue fragments. The meso-dissectioninstrument also consists of an optical imaging system component—with amoving x-y stage to hold the tissue slides where one of the tissueslides, typically a H&E slide, is loaded on to the stage and imaged forthe user to outline the annotations. This slide is referred to asreference slide. The tissue slide to be dissected, thus referred to asmilling slide, is also loaded on the stage and live image captured fortissue extraction. These meso-dissection systems may further provide asoftware interface for enabling annotation of areas of interest andmanually transferring annotations between the images of reference andmilling slides that correspond to serially-cut tissue sections, enablingfurther guidance of the dissection and generation of an electronicrecord of the process.

Although existing meso-dissection systems are more effective than manualdissection methods and are applicable for biomarker analysis ofanatomical pathology samples, they are still deficient in the areas ofcreating the annotations for milling, as the annotations outlined onreference slide are mapped to the milling slide through a manuallyinteractive image alignment and annotation mapping procedure and thesystem does not have the capability to transfer annotations frommultiple reference slides resulting in inaccuracies during scraping offtissue with raw data being corrupted with other forms of tissue versusonly obtaining the tissue from a biologically specific region, and alsonot having the capability to accurately track the biological specimenduring all components of the workflow process from the input slides tothe milled output tissue and any further analysis thereof.

SUMMARY OF THE SUBJECT DISCLOSURE

The subject disclosure solves the above-identified problems by providingsystems, computer-implemented methods, and clinical workflows formeso-dissection of biological specimens and tissue slides includingimporting one or more reference slides with annotations, usinginter-marker registration algorithms to automatically map theannotations to an image of a milling slide, and dissecting the annotatedtissue from the selected regions in the milling slide for analysis,while concurrently tracking the slides, data and analysis using uniqueidentifiers such as bar codes.

In one exemplary embodiment, the subject disclosure provides a system orinstrument for meso-dissection, including a processor, and a memorycoupled to the processor, the memory to store computer-readableinstructions that, when executed by the processor, cause the processorto perform operations comprising importing a reference image along withone or more annotations, wherein the reference image was digitized froma reference slide scanned on a whole-slide scanner and wherein theannotations were generated using a whole slide viewer interface coupledto the whole-slide scanner, and automatically registering said one ormore annotations onto a live capture of a tissue specimen slide to bemilled, wherein tissue is extracted from the tissue specimen slide isdissected based on the one or more annotations, resulting in a milledtissue sample.

In accordance with the embodiment, the operations may further comprisegenerating a milling annotation based on the one or more annotations.The operations may further comprise importing a plurality of annotatedreference images. The milling annotation may further be generated basedon any combination of a plurality of annotations corresponding to theplurality of annotated reference images. The operations may furthercomprise mapping the milling annotation back to the annotated referenceimage. The registering may use an inter-marker registration when thereference slide is stained differently from the tissue specimen slide.The registering may use a same-marker registration when the referenceslide is either stained with the same stain as the tissue specimenslide, or when the reference slide is used as the tissue specimen slide.The one or more annotations may comprise one or more combinations of anygeometrical representation depicting one or more regions of interest.The reference image may be of a different resolution than the livecapture. The instrument may further comprise a tracking means forassociating the one or more annotations with the milled tissue sample.The operations may further comprise associating the one or moreannotations and the milled tissue sample with a biological specimen. Theassociating may comprise using a unique identifier at the specimen leveland additionally for each tissue slide. The operations may furthercomprise providing a user interface to invoke automated registrationalgorithms, logical manipulation of annotations and adjust theregistration of the one or more annotations on the live image.

In another exemplary embodiment, the subject disclosure provides atangible non-transitory computer-readable medium to storecomputer-readable code that is executed by a processor to performoperations comprising generating a milling annotation for milling atissue specimen based on a plurality of reference annotations, andmilling the tissue specimen using the milling annotation, wherein themilling annotation is generated based on automatic registration of anannotation generated on an external whole-slide scanner on to a livecapture of the tissue specimen. The tangible non-transitorycomputer-readable medium may be used in the instrument formeso-dissection as described above.

In yet another exemplary embodiment, the subject disclosure provides atangible non-transitory computer-readable medium to storecomputer-readable code that is executed by a processor to performoperations comprising automatically registering a plurality of referenceannotations with a live-captured image of a tissue sample to bedissected, wherein the plurality of reference annotations are importedfrom a whole-slide scanner, and generating a milling annotation fordissection of the tissue sample based on the results of the automaticregistration. The tangible non-transitory computer-readable medium maybe used in the instrument for meso-dissection as described above.

In yet another exemplary embodiment, the subject disclosure providesmethods performed by said systems. Therefore, the features disclosedwith regard to the systems and computer-readable mediums are understoodto be disclosed with respect to the method, accordingly.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts the main modules of a workflow for tissue analysis usingmeso-dissection, according to an exemplary embodiment of the subjectdisclosure.

FIG. 2 depicts a system for automated meso-dissection, according to anexemplary embodiment of the subject disclosure.

FIG. 3 depicts a method for automated meso-dissection, according to anexemplary embodiment of the subject disclosure.

FIGS. 4A-4B depict tissue and information that is tracked and associatedwith an electronic patient record (EPR), according to an exemplaryembodiment of the subject disclosure.

DETAILED DESCRIPTION OF THE SUBJECT DISCLOSURE

The subject disclosure provides instruments or systems,computer-implemented methods, and clinical workflows for meso-dissectionof biological specimens and tissue slides including importing one ormore reference slides with annotations, using inter-marker registrationalgorithms to automatically map the annotations to an image of a millingslide, and dissecting the annotated tissue from the selected regions inthe milling slide individually or together for analysis, whileconcurrently tracking the data and analysis using unique identifierssuch as bar codes. The tracking is enabled by electronically associatingthe annotations performed on the milling slide, with the milled tissueextracted from the milling slide, and as further described herein.

FIG. 1 depicts the main modules of a workflow 100 for tissue analysisusing meso-dissection. Workflow 100 may utilize a plurality ofsubsystems for performing operations such as a workflow for clinicaldigital pathology. For example, workflow 100 may include an imagingsubsystem 101 for generating an image of an assay or a plurality ofassays. Imaging subsystem 101 may comprise a digital microscope or awhole-slide scanner with imaging components, and may depend on the typeof image being generated. For instance, the sample may have been stainedby means of application of a staining assay containing one or moredifferent biomarkers associated with chromogenic stains for brightfieldimaging or fluorophores for fluorescence imaging, with imaging subsystem101 comprising one or more of a brightfield RGB camera or other capturemechanism, or a fluorescence imaging system. Imaging subsystem 101 mayfurther generate a plurality of images corresponding to serial sectionsof a tissue block, such as a biopsy taken with the intention ofdiagnosing a cancer, or for some other purpose. Each serial section maybe mounted on an individual slide and stained with a differentcombination of stains and biomarkers, resulting in a plurality of imagesdepicting adjacent tissue sections. One such image may be a Hematoxylinand Eosin (H&E) assay. Imaging subsystem 101 may also comprise a cameraattached to a milling subsystem 105, as further described herein.

System 100 may further include an annotation subsystem 103 for enablingselections of annotations of portions of images acquired from imagingsubsystem 101, such as areas or objects of interest. Annotations maybeperformed on, for example, a reference slide image, such as an H&Eslide, so as to indicate areas of interest on the H&E slide forsubsequent scanning, analysis, or registration operations. Annotationsmay be performed automatically, i.e. by detecting regions of interestbased on input from a pathologist, such as by using image analysisoperations, or may be enabled by providing a user interface that depictsone or more images acquired from imaging subsystem 101 to a pathologistor other user via a user interface. For example, annotation subsystem103 may be integrated with an imaging subsystem 101 to form a digitalpathology workstation such as those commercially available today.

A registration subsystem 105 may be invoked to register or map theseannotations to slides of adjacent tissue sections. For example,registration subsystem 105 may be invoked to transfer annotations from areference slide image, such as an H&E slide, to an image of a millingslide, for example a live image of a milling slide captured on ameso-dissection system, such that areas of interest annotated on the H&Eslide are dissected from the milling slide for further analysis.Registration operations may include an ability to register or transferannotations across assays with different combinations of stains andmarkers, including the capability to register an image of any stainedslide to an unstained slide. Such same-marker and inter-markerregistration and annotations methods are further described withreference to commonly-assigned and co-pending EP patent applicationWO2014140070A2, the contents of which are hereby incorporated herein byreference in their entirety. Relevant sections of the incorporatedpatent application describe a computerized image registration processcomprising selecting a first digital image of a first tissue sectionfrom a set of digital images of adjacent tissue sections of a singlepatient, selecting a second digital image of a second tissue sectionfrom the set, matching tissue structure between the first digital imageand the second digital image, and automatically mapping an annotationdrawn on the first digital image to the second digital image. The firstdigital image may be derived from an image obtained using a stain and animaging mode, and the second digital image may be derived from an imageobtained using a different stain, a different imaging mode, or both ascompared to the first digital image. The stain may be chosen from ahematoxylin and eosin stain (‘H&E’ stain), an immunohistochemistry stain(‘IHC” stain), or a fluorescent stain. The imaging mode may be chosenfrom brightfield microscopy or fluorescent microscopy. A matching tissuestructure may comprise a coarse registration mode comprising: generatinga first gray-level tissue foreground image from the first digital imageand generating a second gray-level tissue foreground image from thesecond digital image, computing a first tissue binary edge map from thefirst gray-level tissue foreground image and computing a second tissuebinary edge map from the second gray-level tissue foreground image,computing global transformation parameters to align the first binaryedge map and the second binary edge map, and, mapping the first digitalimage and the second digital image to a common big grid encompassingboth the first and second digital images based on the globaltransformation parameters. Computing global transformation parametersmay further comprise using a moments—based mapping method to generate anaffine mapping between the first binary edge map and the second binaryedge map. A fine registration mode may be used to refine alignment ofthe first digital image and the second digital image. The fineregistration mode comprises: annotating the first digital image, mappingthe annotation on the common big grid to a corresponding location in thesecond digital image, and updating the location using Chamfer—distancematching based on the binary tissue edge maps. Cropped versions of thetissue edge binary maps may be used and the method may further compriseselecting a minimum cost window which improves matching relative tocoarse mode registration.

Another exemplary registration method performed by registrationsubsystem 105 includes a line-based registration operation, includingmodeling the boundary regions of tissue samples reflected in the slideswith line segments, then matching sets of line-segment between tissuesamples (i.e. between slide images) to obtain an overall globaltransformation, i.e. coarse matching. In some embodiments, theline-based coarse matching approach is able to align images even incases of mismatch between images (for example wear-and-tear effects,Area of Interest mismatches which can occur when the area of a physicalslide picked up by the scanner for high-resolution scanning variesbetween adjacent sections, rotation up to 180 degrees, and horizontaland vertical flips), such as when greater than 50% of lines may bematched between the two images. In further embodiments, an additionalfiner sub-image registration process involving normalized,correlation-based, block matching on gradient magnitude images may beexecuted to compute local refinements between globally-aligned images.This registration method is further described with reference tocommonly-assigned and U.S. patent application 61/885,024, the contentsof which are hereby incorporated herein by reference in their entirety.

The registration methods detailed above have the generic capability toregister stained (H&E, IHC etc.) or unstained slides. Any otherregistration method may be used so long as it provides automated mappingof annotations across images having different stains, including thecapability to register stained to unstained tissue slides, in a mannerthat enables precise milling of areas of interest based on annotatedreference slides, or enables registration of stained and unstainedslides. Generally, the subject disclosure also provides an ability toscan a high-resolution reference slide and import it into existingmeso-dissection systems, thereby improving the ability to and transferannotations from high-resolution reference slides to milling annotationsfor a milling slide.

A milling subsystem 107 is used to mill or dissect portions of a tissueslide based on the above-described annotations. Exemplary millingsubsystems are described in commonly-assigned and co-pending patentapplication WO 2012102779A2, the contents of which are herebyincorporated herein by reference in their entirety. In summary, thisreference discloses devices, systems, and associated methods forselectively extracting a material from a sample, wherein in one aspect,for example, a method for selectively extracting a material, such as abiological material, from a sample, such as a biological sample caninclude identifying a region of material to be extracted from a sample,applying an extraction tool to the region of material to disruptmaterial from the sample, and dispensing a liquid at the region ofmaterial. The method can also include aspirating the liquid and thedisrupted material from the sample.

The milling may be performed using a meso-dissection system as describedherein. The meso-dissection system may include an object such as atissue slide attached to a stage capable of controlled X and Y axismovement (i.e. an x-y stage) such that it can be driven against a fixedrotating cutting bit, thereby shaping the object. A plastic mill bit maysimultaneously dispense liquid, cut tissue, and aspirate the tissuefragments from the surface of the glass slide. Because tissue isrelatively soft compared to glass, a spring pressure controlled systemmay be used such that the blade rests on the slide surface withsufficient downward force to cut through the tissue but glides acrossthe glass slide. Milling tissue from glass slides also provides theopportunity to place a digital microscope or camera below the slide inorder to view the process, direct the dissection, and generate digitaldocumentation. The milling subsystem 107 may include software thatprovides an interface to digitally indicate AOIs and save dissectionreference images. Whereas prior art meso-dissection systems merelyimport dissection reference images in standard formats such as .jpg,.png, .bmp, etc. to provide an interface for manually guided alignmentof a single reference image to the live view from a serial tissuesection (milling slide) to enable transferring the areas-of-interest tooverlay the live image and guide the dissection, the subject disclosureprovides automated mapping of annotations from the images of referenceslides to a live-capture of a milling slide using either same-marker orinter-marker registration algorithms to enable more precise dissections.The automated registration and mapping process also makes it easilypossible to transfer annotations from multiple reference images.Moreover, incorporating the annotation and registration operations withthe milling subsystem enables a greater range of tissue slides that maybe dissected, unlike the prior art that is only discloses dissectingunstained tissue. For example, different types of stained assays may besubject to meso-dissection using reference slide annotations, and foreach assay, a coverslip of the assay may be removed prior to dissection,unlike existing systems that only dissect tissue from unstained slidesthat do not already have coverslips attached.

Finally, analysis subsystem 109 performs analysis on the milled tissuesections. This can be any type of analysis, such as molecular (pCR,qTpCR etc.) and/or genetic sequence analysis for clinical and researchpurposes. Results of the analysis may be tracked using trackingmechanisms that link the annotations with the tissue sample or blockthat the reference and milling slides are prepared from, and pre orpost-image analysis and interpretation done on the tissue slides, milledtissue and milled tissue containers, associating these with thebiological specimen being analyzed, and adding the analysis results to adatabase associated with the biological specimen as further describedwith reference to FIG. 4.

FIG. 2 depicts a system 200 for automated meso-dissection, according toan exemplary embodiment of the subject disclosure. System 200 comprisesa memory 210, which stores a plurality of processing modules or logicalinstructions that are executed by processor 215 coupled to computer 216.An input from an imaging or input module 201 may trigger the executionof one or more of the plurality of processing modules. Imaging/inputmodule 201 may include an imaging subsystem such as subsystem 101described with reference to FIG. 1, or may include user inputs as wellas inputs supplied over a network from a network server or database forstorage and later retrieval by computer 216. Besides processor 215 andmemory 210, computer 216 also includes user input and output devicessuch as a keyboard, mouse, stylus, and a display/touchscreen. As will beexplained in the following discussion, processor 215 executes logicalinstructions stored on memory 210, performing importation of referenceimages and annotations from imaging/input module 201 usingannotation/reference importing module 211, transfer or map theannotations from one or more reference images to an image of a tissuesection subject to dissection using registration and mapping module 212,meso-dissection of a milling slide of a tissue sample based on theannotations using milling module 213, and perform tracking of every stepof the process from importation of reference information to output ofmilled tissue into a labeled container for analysis using trackingmodule 214.

Data received from imaging/input 201, or “input data” includes one ormore reference images. The reference images may be scanned at anymagnification level specified by a user, and may depict from any type ofbiological feature, such as H&E or any other IHC, ISH, cytology (urinal,blood smear thin prep, air dried, touch prep, cell block) CTC, and/orhematology slide of interest. The input data may also includeannotations for the imported images. The annotations may have beengenerated on an external digital pathology workstation prior to beinginput into system 200. For example, the annotations may be generated ona whole-slide image management system such as VIRTUOSO® or SCANSCOPE® asa .XML file or annotation data being imported from a server or from adata storage from a network server in the form of JSON or a binary dataor any text format compatible with module 211 and 212 and received byannotation/reference importing module 211. Annotations may be generatedusing image analysis algorithms for detecting and/or segmenting objectsor areas of interest within the reference image(s). Annotations maydepict clinically relevant regions such as immune, tumorous regions ortumor sub-regions where a particular biomarker expression or group ofbiomarker expressions is high. The annotations can be either thecomplete or sub-regions of tumor or any other biologically meaningfulnon-tumor components, such as a lymphatic region, heterogeneous regions,IHC marker sub-type regions, etc. Automated image analysis algorithmsmay detect desired tissue types such as tumors, lymphatic regions in H&Eslide, etc., or hot spots of high marker expression in IHC stainedslides like any tumor, immune or vessel markers tumor markers, immunemarkers, etc.

Besides reference images and annotations, the input data received viaimaging/input module 201 may include information about a target tissuetype or object, as well as an identification of a staining and/orimaging platform. Input information may further include which and howmany specific antibody molecules bind to certain binding sites ortargets on the tissue, such as a tumor marker or a biomarker of specificimmune cells. Additional information input into system 200 may includeany information related to the staining platform, including aconcentration of chemicals used in staining, a reaction times forchemicals applied to the tissue in staining, and/or pre-analyticconditions of the tissue, such as a tissue age, a fixation method, aduration, how the sample was embedded, cut, etc.

Annotation/reference importing module 211 receives the reference imagesand annotations and prepares them to be registered and/or mapped to alive image of a milling slide via registration and mapping module 212.The milling slide may be a tissue slide that is intended to bedissected. For example, the annotations may be imported as a .XML file,or any combination of an XML and image file, enabling the annotations tobe translated to a format that enables mapping the annotations to theimage of the milling slide. The annotation data may include anygeometric information about polygons, rectangles, circles, or ellipses,free hand drawings, labelled region masks, and contours of theannotations, including and excluding regions or areas of interest.Annotations from a plurality of reference images may be imported,enabling multiple areas of interest to be dissected from a millingslide. Annotations drawn at different magnifications and resolutions maybe imported. For instance, annotations drawn on a high resolution (20×or 40×) image may be imported to improve the quality of milling.Annotations may further be generated within system 200 itself by, forinstance, using an automated image analysis algorithm as describedabove. The annotations on the reference slides can be can be eithermanually generated, automatically generated by other image analysisalgorithms or a mixture of manual and algorithm generated. Similarly,the annotations sent for milling on the milling slide can anycombination of annotations a) which are mapped from the reference slide,b) user drawn and c) automatically generated by an image analysisalgorithm. Moreover, the ability to input high-resolution referenceslide, its associated annotations, and import them into the existingmeso-dissection system, enables registering or mapping the imported andany manual or automated edits done to them on the meso-dissection systemannotations to a live image acquired of a milling slide, therebygenerating milling annotations at a resolution appropriate for themilling slide.

Tracking module 214 keeps track of any information associated with thetissue block and tissue slides (like the patient information fromlaboratory information systems (LIMS), clinical or research studyinformation about the biological specimen, staining platform andstaining protocols information, digital scanning information), theimported reference images and corresponding annotations, the millingslide image and the specific milling annotations that are associatedwith specific milling operations performed by milling module 213, milledtissue containers that collect the milled tissue from differentannotated regions individually or together, and molecular and geneticsequence analysis conducted thereof, either individually or incombination and associated reports, as further described with respect toFIG. 4. Prior to milling, registration and mapping module 212 maps theannotations from each reference slide to the live image of the millingslide. This automated registration may use one or more of theinter-marker or same-marker registration methods described herein and inthe references incorporated above. For instance, a same-markerregistration method is used if the reference and milling slide are thesame slide or of the same stain type. Importing reference images andannotations eliminates the need to load a reference slide into animaging platform of existing meso-dissection systems. Only the imagedata and annotations may be imported from an external system. Moreover,universal importation of annotations from differently-stained referenceslides enables meso-dissection of not only unstained tissue slides, butalso stained tissue slides, once the coverslip is removed. In someembodiments of the subject disclosure, the same tissue that is used as areference image may also be used for milling purposes. Additionally, theability to import multiple reference images along with their annotationsenables mapping of annotations from multiple reference images to asingle or multiple milling slides for tissue dissection. For example,given a tissue block from a patient where multiple individual biomarkerslides such as H&E, ER, PR, HER2, ISH, etc. may be digitized andannotated individually, any desired combination of these images andassociated annotations can be imported and automatically registered andmapped to the image of the milling slide. In addition to mappingannotations from reference images to milling slide images, registrationand mapping module 212 may provide an interface to fine-tune or adjustthe annotations prior to milling.

Milling module 213 performs dissection on the milling slides byfollowing the annotations mapped from the reference image(s) to the liveimage of the milling slide. The live image of the milling slide may becaptured by another imaging device (not shown) in communication withmilling module 213. Milling module 213 may further determine thenecessary metrics for milling, including determining an optimal millingpath on the milling slide based on the annotations, as well asdetermining a required diameter of the pipette and optimal volume ofliquid to be dispensed for the tissue intended for dissection. Thisautomated milling based on milling annotations generated from theregistration and mapping of the reference annotations reduces the burdenof a technician or human operator manually annotating the milling slideimage. Moreover, to enable flexible functionality and for differentpurposes, milling module 213 may invoke registration/mapping module 212to map back the milling annotations (i.e. the regions that are dissectedfrom the milled slides) to the reference slides. These milled regionscan be clearly indicated and graphically overlaid on each of thereference images. The mapped milled annotations may be used to assistwith selection of additional regions for tissue analysis or annotationof more regions for further milling. Such mapping back may be enabled bystoring the milled annotations to memory 210, and retrieving andregistering back to the reference slides already used or for any otheradditional set of tissue slides from adjacent serial sections. Themilled annotations are specified on the live image, thus in the cameracoordinate system of the meso-dissection system, i.e. the same cameraused to capture the live image of the milling slide, in contrast to thecoordinate system of the camera originally used to generate thereference images. Any adjustments that are performed to the millingannotations prior to or during the milling process may be registeredback to the original reference slide(s). Therefore, the annotation andregistration across platforms is improved. Moreover, the slide beingmilled may be the same as the slide used to generate the referenceimage, enabling the annotations from milling to be updated back to thereference image annotations, enabling a more accurate annotation on thereference image. To map back the milling annotations from the milledimage to each of the reference images, the automated registrationoperations described herein may be invoked. The milled image and themilled annotations are also part of tracked information.

As described above, the modules include logic that is executed byprocessor 105. “Logic”, as used herein and throughout this disclosure,refers to any information having the form of instruction signals and/ordata that may be applied to affect the operation of a processor.Software is one example of such logic. Examples of processors arecomputer processors (processing units), microprocessors, digital signalprocessors, controllers and microcontrollers, etc. Logic may be formedfrom signals stored on a computer-readable medium such as memory 210that, in an exemplary embodiment, may be a random access memory (RAM),read-only memories (ROM), erasable/electrically erasable programmableread-only memories (EPROMS/EEPROMS), flash memories, etc. Logic may alsocomprise digital and/or analog hardware circuits, for example, hardwarecircuits comprising logical AND, OR, XOR, NAND, NOR, and other logicaloperations. Logic may be formed from combinations of software andhardware. On a network, logic may be programmed on a server, or acomplex of servers. A particular logic unit is not limited to a singlelogical location on the network. Moreover, the modules need not beexecuted in any specific order. Each module may call another module whenneeded to be executed.

FIG. 3 depicts a method for automated meso-dissection, according to anexemplary embodiment of the subject disclosure. The method of FIG. 3 maybe executed by any combination of the modules depicted in FIG. 2, or anyother combination of modules. The steps listed in this method need notbe executed in the particular order shown. The method may begin with aninput S301 of any combination of: a biological question/requesteddiagnosis by a pathologist, patient or biological specimen informationenabling tracking, and annotated (or non-annotated) reference images oftissue sections associated with the tissue specimen (tissue block,patient, xenograft, etc). The reference images may be scanned at anymagnification level specified by a pathologist, and may depict from anytype of biological feature, such as H&E or any other IHC, ISH, cytology(urinal, blood smear thin prep, air dried, touch prep, cell block) CTC,and/or hematology slide of interest. The input data may also includeannotations for the reference images.

Annotations of input reference images are imported (S302). Theannotations may have been generated on an external digital pathologyworkstation prior to being imported. Annotations may be generated usingimage analysis algorithms for detecting and/or segmenting objects orareas of interest within the reference image(s). Annotations may depictclinically relevant regions such as immune or tumor regions where theparticular biomarker expression is high. The annotations can be eitherthe complete or sub-regions of tumor or any other biologicallymeaningful non-tumor components, such as a lymphatic region,heterogeneous regions, IHC marker sub-type regions, etc. Automated imageanalysis algorithms may detect desired tissue types such as tumors,lymphatic regions in H&E slide, etc., or hot spots of high markerexpression in IHC stained slides like any tumor, immune or vesselmarkers tumor markers, immune markers, etc. The annotations may beprepared to be registered and/or mapped to a live image of a millingslide. For example, the annotations may be imported as a .XML file, orany combination of an XML and image file, enabling the annotations to betranslated to a format that enables mapping the annotations to the imageof the milling slide.

Annotations from a plurality of reference images may be imported,enabling multiple areas of interest to be dissected from a millingslide. For example, the annotations for milling can be generated from apermutation or combination of annotations specified on one or multiplereference slides. The milling annotations may be user-specified, tissuetype specific, and problem specific—i.e. generated to answer aparticular biological question (like tumor heterogeneity, IHC4, immunescoring, tumor region analysis, peri-tumoral regional analysis,evaluation of tumor microenvironment, etc.) in a particular tissue typesuch as breast cancer, lung cancer, prostate cancer, colorectal canceretc. For example, if co-expression analysis is important, the individualmarker annotations are specified on each of the marker reference slides.In this case, the milling annotations, to indicate the specific regionsfor co-expression analysis, are a logical intersection of theseannotations from multiple individual slide annotations. Similarly, ifthe biological interest is an inclusion of all the annotated regionsfrom all markers, the milling annotations are a logical union of all theindividual reference slide(s) annotations. For example, for a breastcancer patient, a typical series of IHC slides includes an H&E slide, aER marked slide, a KR and PR marked slide, a Ki-67 marked slide, and anHER2-marked slide, each one having separate annotations for theexpression of their respective markers. In the H&E slide, for instance,tumor and lymphatic regions may be annotated. A pathologist or biologistmay be interested in a region that is ER positive and PR negative, orwhere all the markers are PR positive and Ki-67 negative, or any logicalcombination of regions for analysis. Multiple combinations of theseconstraints may be specified in order to automatically generate millingannotations. In another example, given an H&E slide associated with atissue block, an adjacent serial section of a BRAFV600E-stained IHCslide, and an adjacent serial section PTEN-stained IHC slide, and abiological question to identify regional areas where there is expressionof BRAFV600E and no expression of PTEN, the method can respectivelyidentify the BRAFV600E-expressing regions (from the BRAFV600E slide) andPTEN non-expressing regions (from the PTEN slide), and identify regionswhere these areas overlap on the H&E slide. A resultant millingannotation would include these overlapping areas to be meso-dissected onthe H&E slide. Any combinations of tumor markers or a combination ofexpressions (such as positive/negative) may be specified to furtherunderstand the tumor environment of the tissue specimen. Thus, asdictated by the underlying biological problem, several logicalpermutations and combinations of annotations can be used to constructthe set of milling annotations. The milling annotations are the onesthat get used by a milling system to extract the tissue from each of themilling slides into the specified tissue containers as further describedherein. Moreover, annotations drawn at a higher magnification levelsimported with the reference slides and used to generate the millingannotations ensure that a higher-quality milling operation is performedwith minimal corruption of raw data (milled tissue) for analysis.

A milling slide is set up on a milling stage or platform, and imaged(S303) using a camera 314 coupled to the milling stage or platform. Themilling image is typically a live capture. The milling slide may be anytissue slide, either unstained, or stained, and may be the same as thetissue slide used to generate one or more of the reference images.Annotations imported in step S302 may be registered (S304) and mapped tothe live image of the milling slide. This automated registration may useone or more of the inter-marker or same-marker registration methodsdescribed herein and in the references incorporated above. For instance,a same-marker registration method is used if the reference and millingslide are the same slide or of the same stain type. Moreover, universalimportation of annotations from differently-stained reference slidesenables meso-dissection of not only unstained tissue slides, but alsostained tissue slides, once the coverslip is removed. In someembodiments of the subject disclosure, the same tissue that is used as areference image may also be used for milling purposes. Additionally, theability to import multiple reference images along with their annotationsenables mapping of annotations from multiple reference images to asingle or multiple milling slides for tissue dissection. For example,given a tissue block from a patient where multiple individual biomarkerslides such as H&E, ER, PR, HER2, ISH, etc. may be digitized andannotated individually, any desired combination of these images andassociated annotations can be imported and automatically registered andmapped to the image of the milling slide. In addition to mappingannotations from reference images to milling slide images, an interfacemay be provided to fine-tune or adjust the annotations prior to milling.

A container may be prepared (S305) to receive the milled tissue from themilling operation. The container may be labeled with an identifierunique to a record of the biological specimen under diagnosis, enablingtracking of the milled tissue along with association of the milledtissue with the particular annotation and reference slide. This ensuresthat each imported reference image and annotation is properly associatedwith specific milling operations and resulting milled tissue, all ofwhich may be associated with an electronic patient record. The millingslide may be dissected (S306) based on the milling annotations, usingmetrics such as an optimal milling path on the milling slide or arequired diameter of the pipette and optimal volume of liquid to bedispensed for the tissue intended for dissection, based on the millingannotations. As described above, several logical permutations andcombinations of input annotations can be used to construct the set ofmilling annotations that are used by milling operation (S306) to drilland extract the tissue from the milling slide into the specified tissuecontainer or a set a tissue containers. The dissected tissue may bemilled into the particular container labeled in association with theparticular annotation to ensure proper tracking of the tissue and theassociated analysis thereof.

Any combination of milling operations performed on the milling slide maybe converted into annotations and mapped back (S307) into one or more ofthe reference slides. For example, an annotation from a first referenceslide may be used to mill the milling slide, and the same annotation maybe mapped back to a second reference slide, or to the same referenceslide. Combinations of annotations from multiple reference slides may bemilled and subsequently mapped back to a single reference slide. Theupdated reference slide annotations may be used to assist with selectionof additional regions for tissue analysis or annotation of more regionsfor further milling. Such mapping back may be enabled by tracking andstoring the milled annotations and retrieving and registering back tothe reference slides already used or for any other additional set oftissue slides from adjacent serial sections. The method ends (S309) byunloading the tracked container with the milled tissue.

FIG. 4A depicts information that is tracked and associated with anelectronic patient record (EPR), according to an exemplary embodiment ofthe subject disclosure. As described herein, patient data andinformation may be tracked by bar-coding (or using an equivalent uniquelabeling system) every component of the above-described workflows andrecording them in an EPR 440. For example, a reference slide and itscorresponding annotations 441 is associated with EPR 440. The millingslide 442 that corresponds to the same serial section as the referenceslide 441 is also associated with EPR 440. Importantly, the container443 that is used to receive the milled tissue from milling slide 442 istracked using the same bar code or as associated set of bar codes, andany resulting analysis data 444 of the milled tissue is also trackedwith the same bar code. Therefore, a single EPR 440 maintains allanalytic and process information that is enhanced using the accurateannotation registering and milling operations described herein. FIG. 4Bdepicts a barcoded container of extracted or dissected tissue sample.Although barcodes are described herein, other methods for trackingand/or tagging may be used, such as RFID, etc.

The disclosed operations therefore mitigate tedious manual transcriptionof annotations on existing milling systems, while avoiding errors basedon mixing up or corrupting tissue-specific analysis data, thereforemaking the overall system robustly operational in a high-volume usage inpre-clinical and clinical settings and in drug discovery and drugdevelopment experimental studies. Moreover, besides medical applicationssuch as anatomical or clinical pathology, prostrate/lung cancerdiagnosis, etc., the same methods may be performed to analysis othertypes of samples such as remote sensing of geologic or astronomicaldata, etc. The operations disclosed herein may be ported into a hardwaregraphics processing unit (GPU), enabling a multi-threaded parallelimplementation.

Computers typically include known components, such as a processor, anoperating system, system memory, memory storage devices, input-outputcontrollers, input-output devices, and display devices. It will also beunderstood by those of ordinary skill in the relevant art that there aremany possible configurations and components of a computer and may alsoinclude cache memory, a data backup unit, and many other devices.Examples of input devices include a keyboard, a cursor control devices(e.g., a mouse), a microphone, a scanner, and so forth. Examples ofoutput devices include a display device (e.g., a monitor or projector),speakers, a printer, a network card, and so forth. Display devices mayinclude display devices that provide visual information, thisinformation typically may be logically and/or physically organized as anarray of pixels. An interface controller may also be included that maycomprise any of a variety of known or future software programs forproviding input and output interfaces. For example, interfaces mayinclude what are generally referred to as “Graphical User Interfaces”(often referred to as GUI's) that provide one or more graphicalrepresentations to a user. Interfaces are typically enabled to acceptuser inputs using means of selection or input known to those of ordinaryskill in the related art. The interface may also be a touch screendevice. In the same or alternative embodiments, applications on acomputer may employ an interface that includes what are referred to as“command line interfaces” (often referred to as CLI's). CLI's typicallyprovide a text based interaction between an application and a user.Typically, command line interfaces present output and receive input aslines of text through display devices. For example, some implementationsmay include what are referred to as a “shell” such as Unix Shells knownto those of ordinary skill in the related art, or Microsoft WindowsPowershell that employs object-oriented type programming architecturessuch as the Microsoft .NET framework.

Those of ordinary skill in the related art will appreciate thatinterfaces may include one or more GUI's, CLI's or a combinationthereof. A processor may include a commercially available processor suchas a Celeron, Core, or Pentium processor made by Intel Corporation, aSPARC processor made by Sun Microsystems, an Athlon, Sempron, Phenom, orOpteron processor made by AMD Corporation, or it may be one of otherprocessors that are or will become available. Some embodiments of aprocessor may include what is referred to as multi-core processor and/orbe enabled to employ parallel processing technology in a single ormulti-core configuration. For example, a multi-core architecturetypically comprises two or more processor “execution cores”. In thepresent example, each execution core may perform as an independentprocessor that enables parallel execution of multiple threads. Inaddition, those of ordinary skill in the related will appreciate that aprocessor may be configured in what is generally referred to as 32 or 64bit architectures, or other architectural configurations now known orthat may be developed in the future.

A processor typically executes an operating system, which may be, forexample, a Windows type operating system from the Microsoft Corporation;the Mac OS X operating system from Apple Computer Corp.; a Unix orLinux-type operating system available from many vendors or what isreferred to as an open source; another or a future operating system; orsome combination thereof. An operating system interfaces with firmwareand hardware in a well-known manner, and facilitates the processor incoordinating and executing the functions of various computer programsthat may be written in a variety of programming languages. An operatingsystem, typically in cooperation with a processor, coordinates andexecutes functions of the other components of a computer. An operatingsystem also provides scheduling, input-output control, file and datamanagement, memory management, and communication control and relatedservices, all in accordance with known techniques.

System memory may include any of a variety of known or future memorystorage devices that can be used to store the desired information andthat can be accessed by a computer. Computer readable storage media mayinclude volatile and non-volatile, removable and non-removable mediaimplemented in any method or technology for storage of information suchas computer readable instructions, data structures, program modules, orother data. Examples include any commonly available random access memory(RAM), read-only memory (ROM), electronically erasable programmableread-only memory (EEPROM), digital versatile disks (DVD), magneticmedium, such as a resident hard disk or tape, an optical medium such asa read and write compact disc, or other memory storage device. Memorystorage devices may include any of a variety of known or future devices,including a compact disk drive, a tape drive, a removable hard diskdrive, USB or flash drive, or a diskette drive. Such types of memorystorage devices typically read from, and/or write to, a program storagemedium such as, respectively, a compact disk, magnetic tape, removablehard disk, USB or flash drive, or floppy diskette. Any of these programstorage media, or others now in use or that may later be developed, maybe considered a computer program product. As will be appreciated, theseprogram storage media typically store a computer software program and/ordata. Computer software programs, also called computer control logic,typically are stored in system memory and/or the program storage deviceused in conjunction with memory storage device. In some embodiments, acomputer program product is described comprising a computer usablemedium having control logic (computer software program, includingprogram code) stored therein. The control logic, when executed by aprocessor, causes the processor to perform functions described herein.In other embodiments, some functions are implemented primarily inhardware using, for example, a hardware state machine. Implementation ofthe hardware state machine so as to perform the functions describedherein will be apparent to those skilled in the relevant arts.Input-output controllers could include any of a variety of known devicesfor accepting and processing information from a user, whether a human ora machine, whether local or remote. Such devices include, for example,modem cards, wireless cards, network interface cards, sound cards, orother types of controllers for any of a variety of known input devices.Output controllers could include controllers for any of a variety ofknown display devices for presenting information to a user, whether ahuman or a machine, whether local or remote. In the presently describedembodiment, the functional elements of a computer communicate with eachother via a system bus. Some embodiments of a computer may communicatewith some functional elements using network or other types of remotecommunications. As will be evident to those skilled in the relevant art,an instrument control and/or a data processing application, ifimplemented in software, may be loaded into and executed from systemmemory and/or a memory storage device. All or portions of the instrumentcontrol and/or data processing applications may also reside in aread-only memory or similar device of the memory storage device, suchdevices not requiring that the instrument control and/or data processingapplications first be loaded through input-output controllers. It willbe understood by those skilled in the relevant art that the instrumentcontrol and/or data processing applications, or portions of it, may beloaded by a processor, in a known manner into system memory, or cachememory, or both, as advantageous for execution. Also, a computer mayinclude one or more library files, experiment data files, and aninternet client stored in system memory. For example, experiment datacould include data related to one or more experiments or assays, such asdetected signal values, or other values associated with one or moresequencing by synthesis (SBS) experiments or processes. Additionally, aninternet client may include an application enabled to access a remoteservice on another computer using a network and may for instancecomprise what are generally referred to as “Web Browsers”. In thepresent example, some commonly employed web browsers include MicrosoftInternet Explorer available from Microsoft Corporation, Mozilla Firefoxfrom the Mozilla Corporation, Safari from Apple Computer Corp., GoogleChrome from the Google Corporation, or other type of web browsercurrently known in the art or to be developed in the future. Also, inthe same or other embodiments an internet client may include, or couldbe an element of, specialized software applications enabled to accessremote information via a network such as a data processing applicationfor biological applications.

A network may include one or more of the many various types of networkswell known to those of ordinary skill in the art. For example, a networkmay include a local or wide area network that may employ what iscommonly referred to as a TCP/IP protocol suite to communicate. Anetwork may include a network comprising a worldwide system ofinterconnected computer networks that is commonly referred to as theinternet, or could also include various intranet architectures. Those ofordinary skill in the related arts will also appreciate that some usersin networked environments may prefer to employ what are generallyreferred to as “firewalls” (also sometimes referred to as PacketFilters, or Border Protection Devices) to control information traffic toand from hardware and/or software systems. For example, firewalls maycomprise hardware or software elements or some combination thereof andare typically designed to enforce security policies put in place byusers, such as for instance network administrators, etc.

In one embodiment, disclosed is a tangible non-transitorycomputer-readable medium to store computer-readable code that isexecuted by a processor to perform operations comprising: automaticallyregistering a plurality of reference annotations with a live-capturedimage of a tissue sample to be dissected, wherein the plurality ofreference annotations are imported from a whole-slide scanner; andgenerating a milling annotation for dissection of the tissue samplebased on the results of the automatic registration.

In another embodiment, disclosed is a computer-implemented method formeso-dissection comprising the steps: importing a reference image alongwith one or more annotations, wherein the reference image was digitizedfrom a reference slide scanned on a whole-slide scanner and wherein theannotations were generated using a whole slide viewer interface coupledto the whole-slide scanner; and automatically registering said one ormore annotations onto a live capture of a tissue specimen slide to bemilled; wherein tissue is extracted from the tissue specimen slide isdissected based on the one or more annotations, resulting in a milledtissue sample.

In yet another embodiment, disclosed is a computer-implemented methodfor meso-dissection comprising the steps: automatically registering aplurality of reference annotations with a live-captured image of atissue sample to be dissected, wherein the plurality of referenceannotations are imported from a whole-slide scanner; and generating amilling annotation for dissection of the tissue sample based on theresults of the automatic registration.

What is claimed is:
 1. An instrument for meso-dissection, comprising: aprocessor; and a memory coupled to the processor, the memory to storecomputer-readable instructions that, when executed by the processor,cause the processor to perform operations comprising: importing areference image along with one or more reference annotations, whereinthe reference image was digitized from a reference slide scanned on awhole-slide scanner and wherein the one or more reference theannotations were generated using a whole slide viewer interface coupledto the whole-slide scanner; automatically registering the one or morereference annotations onto a live capture of a tissue specimen slide tobe milled; generating a milling annotation based on the one or morereference annotations; dissecting tissue from the tissue specimen slidebased on the generated milling annotation to provide, a milled tissuesample; automatically associating the one or more reference annotationsand the milling annotation with the milled tissue sample and with abiological specimen; and mapping the milling annotation back to theannotated reference image.
 2. The instrument of claim 1, wherein theoperations further comprise importing a plurality of annotated referenceimages.
 3. The instrument of claim 2, wherein the milling annotation isgenerated based on any combination of a plurality of annotationscorresponding to the plurality of annotated reference images.
 4. Theinstrument of claim 1, wherein registering comprises using aninter-marker registration when the reference slide is staineddifferently from the tissue specimen slide.
 5. The instrument of claim1, wherein registering comprises using a same-marker registration whenthe reference slide is either stained with the same stain as the tissuespecimen slide, or when the reference slide is used as the tissuespecimen slide.
 6. The instrument of claim 1, wherein the one or moreannotations comprise one or more combinations of any geometricalrepresentation depicting one or more regions of interest.
 7. Theinstrument of claim 1, wherein the reference image is of a differentresolution than the live capture.
 8. The instrument of claim 1, whereinthe associating comprises using a unique identifier at the specimenlevel and additionally for each tissue slide.
 9. The instrument of claim1, wherein the operations further comprise providing a user interface toinvoke automated registration algorithms, logical manipulation ofannotations and adjust the registration of the one or more annotationson the live image.
 10. A tangible non-transitory computer-readablemedium to store computer-readable code that is executed by a processorto perform operations comprising: generating a milling annotation formilling a tissue specimen based on a plurality of reference annotations;and milling the tissue specimen using the milling annotation, whereinthe milling annotation is generated using an automated intermarkerregistration algorithm, and wherein the automated intermakerregistration algorithm registers an annotation within a whole slideimage generated on an external whole-slide scanner on to a live captureof the tissue specimen.
 11. The tangible non-transitorycomputer-readable medium of claim 10, wherein the operations furthercomprise automatically associating the one or more annotations with themilled tissue sample and with a biological specimen.
 12. Thecomputer-implemented method of claim 10, wherein the milling tissue isstained differently than the whole slide image.
 13. Acomputer-implemented method for meso-dissection comprising the steps:generating a milling annotation for milling a tissue specimen based on aplurality of reference annotations; and milling the tissue specimenusing the milling annotation, wherein the milling annotation isgenerated using an automated line-based registration algorithm, andwherein the automated line-based registration algorithm registers anannotation within a whole slide image generated on an externalwhole-slide scanner on to a live capture of the tissue specimen.
 14. Thecomputer-implemented method of claim 13, wherein the operations furthercomprise automatically associating the one or more annotations with themilled tissue sample and with a biological specimen.
 15. Thecomputer-implemented method of claim 13, wherein the line-basedregistration comprises modeling boundary regions of the tissue specimenswith line segments, then matching sets of the line segments between thetissue specimens.